Corpus-based unit selection for natural-sounding speech synthesis

Speech synthesis is an automatic encoding process carried out by machine through which symbolsconveying linguistic information are converted into an acoustic waveform. In the past decade orso, a recent trend toward a non-parametric, corpus-based approach has focused on using real hu-man speech as source material for producing novel natural-sounding speech. This work proposes acommunication-theoretic formulation in which unit selection is a noisy channel through which aninput sequence of symbols passes and an output sequence, possibly corrupted due to the coveragelimits of the corpus, emerges. The penalty of approximation is quantified by substitution and con-catenation costs which grade what unit contexts are interchangeable and where concatenations arenot perceivable. These costs are semi-automatically derived from data and are found to agree withacoustic-phonetic knowledge. The implementation is based on a finite-state transducer (FST) representation that has been success-fully used in speech and language processing applications including speech recognition. A proposedconstraint kernel topology connects all units in the corpus with associated substitution and con-catenation costs and enables an efficient Viterbi search that operates with low latency and scales tolarge corpora. An A∗ search can be applied in a second, rescoring pass to incorporate finer acousticmodelling. Extensions to this FST-based search include hierarchical and paralinguistic modelling.The search can also be used in an iterative feedback loop to record new utterances to enhance corpuscoverage. This speech synthesis framework has been deployed across various domains and languages in manyvoices, a testament to its flexibility and rapid prototyping capability. Experimental subjects com-pleting tasks in a given air travel planning scenario by interacting in real time with a spoken dialoguesystem over the telephone have found the system “easiest to understand” out of eight competing sys-tems. In more detailed listening evaluations, subjective opinions garnered from human participantsare found to be correlated with objective measures calculable by machine. Thesis Supervisor: James R. GlassTitle: Principal Research Scientist